Systems and Methods for Automated Extraction of Closed Captions in Real Time or Near Real-Time and Tagging of Streaming Data for Advertisements

ABSTRACT

System and methods for finding and analyzing target content from audio and video content sources, including means and methods for extracting captions from audio and video content sources; searching the captions for a mention of at least one target; extracting audio and video segments relating to the at least one target; delivering extracted audio and video segments to a user device; harvesting social media data relevant to the target content; analyzing the search results in correlation with the social media data for target content.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application relates to and claims priority from the following U.S.Applications. This application is a continuation-in-part of U.S.application Ser. No. 15/049,376 filed Feb. 22, 2016, which is acontinuation of U.S. application Ser. No. 14/711,257 filed on May 13,2015, which is a continuation of U.S. application Ser. No.14/299,833filed on Jun. 9, 2014, which is a continuation of U.S. application Ser.No. 13/834,290 filed on Mar. 15, 2013, which is a continuation-in-partof U.S. application Ser. No. 12/967,135 filed on Dec. 14, 2010, whichclaims the benefit of U.S. Application Ser. No. 61/287,868 filed on Dec.18, 2009, each of which is incorporated herein by reference in itsentirety.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates generally to electronic data streamingmanagement. Further, the present invention relates to automated realtime or near real time extraction of closed captions systems and methodsrelating thereto, and more particularly to advertisement video closedcaptioning. Near real-time extraction is extraction that is initiatedduring the broadcast whereas real-time extraction has no time delay.

2. Description of the Prior Art

Many TV broadcasts and owners of video content enable users to seecomplete segments of a TV program or partial segments of a TV program.While this is usually sufficient for an individual with access to a highspeed device, the individual can only ‘see’ and ‘listen’ to one channelat a time and sometimes to several. There is no ability for anindividual or organization to monitor in real time dozens or hundreds ofTV channels for particular keywords, concepts or phrases (words ofinterest (WOI)) and be alerted of the occurrence of those words, browsefor WOI, search to any WOI and persist the WOI over a long period oftime. Further, there is a need to be able to deliver the WOI over alower speed network, such as a telephone provider's network, inincreasing bandwidth without clogging the network infrastructure.

There also remains a need for automated systems and methods for encodingand embedding tag(s) associated with data streams for providing searchcapability of the data streams.

Relevant prior art U.S. patents and published pending U.S. patentapplications include the following:

U.S. Pat. Nos. 5,859,662, 5,481,296, 6,266,094 and U.S. Pub. Nos.2008/0313146 and 2003/0221198 relate to extracting captions from videobroadcasts.

U.S. Pat. No. 5,859,662 and U.S. Pub. No. 2009/0049481 relate toextracting captions in real time.

U.S. Pat. No. 7,518,657 relates to storing captions on a device orpushing to a cloud.

U.S. Pat. Nos. 5,859,662 and 6,266,094 relate to providing alerts basedupon key words.

U.S. Pat. No. 5,481,296 relates to providing alerts based upon conceptsof interest.

U.S. Pat. Nos. 6,580,437 and 6,798,912 relate to creating an index ofvideo segments based upon caption information.

U.S. Pat. Nos. 5,859,662, 7,467,398, 5,561,457 and U.S. Pub. Nos.2007/0027844, 2008/0313146, 2009/0049481 and 2003/0093814 relate toviewing indexed video or audio based on caption searches.

U.S. Pat. Nos. 5,859,662, 5,481,296 relate to software for an end userfor the related technology; and U.S. Pub. No. 2003/0192050 relates tosoftware for broadcast location.

U.S. Pat. No. 6,457,010 relates to storing information about a user'sprofile.

U.S. Pat. No. 7,210,157 and U.S. Pub. Nos. 2007/0300250 and 2003/0221198relate to allowing for finding media based on a user's profile.

In addition to the patent references listed hereinabove, it is known inthe art to provide for free and licensed applications that allowindividuals to record and extract CC of completed recordings. Theseapplications are typically located at the end-user's premise, providefor a limited number of channel recordings and provide limited databaseand search capabilities. Most of these applications are aimed atproviding traditional Personal Video Recording (PVR) functionality suchas record this program at this time on this channel. Some allow foradded features such as limited keyword searches of extracted captionsand only of recordings in the format of the vendor of the TV tuner. Allof them enable extraction after the broadcasts have been recorded andnot while the broadcasts are in progress.

Open-Source close caption (CC) extraction applications include:

a) The SCC Tools package consists of ten command-line tools (and oneGeneral Parser module) designed to assist in the task of extracting,manipulating, and inserting the additional data included in Line 21 ofNTSC video: closed captions, MSNTV links, V-Chip ratings, and a varietyof lesser-used types of information.http://www.geocities.com/mcpoodle43/SCC_TOOLS/DOCS/SCC_TOOLS.HTML#CCExtract

b) MPG2SRT-MPG2SRT is a standalone program to extract closed captioningdata embedded within an MPEG2 file. The extracted captions can be savedin a .srt format for use with directvobsub or similar application, or asa .SAMI file for use with Windows Media Player.http://www.htpctools.com/mpg2srt/

c)http://ccextractor.sourceforge.net/ccextractor_for_windows.html

There are companies that provide (fee or free) PVR or DVR functionalitysoftware. All provide the basic and/or enhanced PVR or DVR capabilitiesand some provide extended capabilities. Some features include: pause,rewind, fast-forward live; record all favorite TV shows by name; andintegrated TV guide (provided by the DVR software). Some companiesproviding commercially available products or services at the time of thepresent invention include:

a) Microsoft MediaCenter—allows for recording of selected channels atparticular times and all of the traditional PVR functionality.

b) SnapStream Personal and Enterprise edition products are the mostadvanced. The products are able to capture, index and extract captionsand alert users based on keywords. The application is aimed at anindividual (personal device) or an organization and is bundled with itsown hardware and software. A full description of the capability of thedevice is: http://www.snapstream.com/enterprise/features.asp.

c) ATI Multi-Media Center(http://ati.amd.com/products/multimediacenter/features.html) allows theuser to record and search the Closed Caption text during TV-on-Demand™sessions and is limited to the number of tuners in the user's system.Creation and delivery of alerts are limited.

d) SageTV (http://sagetv.com/stvfeatures.html?sageSub=tv) offers manyfeatures of an advanced PVR and DVR.

e) MythTV (www.mythtv.org) is a Free Open Source software digital videorecorder (DVR) project distributed under the terms of the GNU GPL. Ithas been under heavy development since 2002, and now contains mostfeatures one would expect from a good DVR.

SUMMARY OF THE INVENTION

A first aspect of the present invention is to provide methods andsystems to extract in real time or near real-time captions from Videobroadcasts that have Closed Captions (CC), extract encoded nearreal-time advertisements, provide alerts based on keywords or conceptsof interest, extract parts or entire audio from a video broadcast,search captions and enable users to index into the video or audiosegments that are relevant to the captions, view or listen to the searchresults, assemble a ‘personal’ audio and video of the results into apersonalized clip and run the environment in a distributed orcentralized manner as a dedicated environment or a service environment.This capability can be in a general or dedicated device such as a PC orembedded in a device such as a TV tuner, PVR or DVR or any intelligentcomputing device, including SOC and mobile devices. Near real-timeextraction is extraction that is initiated during the broadcast whereasreal-time extraction has no time delay.

A second aspect of the present invention is to provide systems andmethods to encode and embed a stream of bits that represent anAdvertisement Tag Code (ATC) for providing automatic electronic methodsfor collecting data about at least one ATC and correlate collected datawith additional sources of data. The ATC may be encrypted orunencrypted. Benefits of methods and system of the present invention forapplications in advertising include providing a campaign managercapabilities to monitor automatically and electronically theeffectiveness of a particular advertising campaign, the occurrence of‘earned media’ relevant to the campaign and to correlate such campaignwith traditional print media, internet media, social media and mobilemedia campaigns. The ATC is placed in the VBI or closed captioned streamof a broadcast TV channel, or in a live Internet video stream.

While other systems exist for tagging advertisements such as Ad-ID, thepresent system and method provide for an open and widely availableservice that does not rely on a central authority to design anddistribute the advertisement TAG for any content. For example, an ATCmay be inserted into a data stream to enable users to automatically linkto a company's web site for a particular product or particular campaign.Such an ATC would therefore facilitate the integration of any contentfrom the live broadcast to any other content (web logs, web pages, phonelogs, etc.) for the purpose of producing deeper analytics about theeffectiveness of the message; whether it is ad campaign-related orotherwise.

These and other aspects of the present invention will become apparent tothose skilled in the art after a reading of the following description ofthe preferred embodiment when considered with the drawings, as theysupport the claimed invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a schematic view of one embodiment of the presentinvention.

FIG. 2 illustrates a schematic view of another embodiment of the presentinvention.

FIGS. 3-12 illustrate screen shots of various graphic user interfacesfor an interactive website according to one embodiment of the presentinvention.

FIG. 3 shows an interface for retrieving captions for a specific show.

FIG. 4 shows an interface for retrieving captions for a specific dateand time for the show selected in FIG. 3.

FIG. 5 shows an interface for searching shows.

FIG. 6 shows an interface for displaying query results.

FIG. 7 shows an interface for searching for advertisementssubscriptions.

FIG. 8 shows an interface for displaying query results in a timesequence.

FIG. 9 shows an interface for displaying query results by show withcount, including a graph display.

FIG. 10 is the graph of FIG. 9, enlarged for better visibility ofdetails.

FIG. 11 is an interface for displaying query results with a table andgraph showing query hits and count according to channel.

FIG. 12 is another table and graph showing query hits and countaccording to channel.

DETAILED DESCRIPTION

Referring now to the drawings in general, the illustrations are for thepurpose of describing a preferred embodiment of the invention and arenot intended to limit the invention thereto.

Near Real-Time Extraction

The present invention also provides methods and systems for extractingnear real-time captions from Video broadcasts that have Closed Captions(CC), extract in near real-time encoded advertisements or targetedcontent, provide alerts based on keywords or concepts of interest,extract parts or entire audio from a video broadcast, search captionsand enable users to index into the video or audio segments that arerelevant to the captions and run the environment in a distributed orcentralized manner as a dedicated environment or a service environment.

The captions are extracted from any video and audio broadcast (TV orInternet) and inserted into a database that allows for alerting based onWOI, searching, indexing into video and audio segments and extraction ofall or partial audio from the video broadcast. There is a limited amountof bandwidth available on mobile devices, and lower speed networkedconnection, and this invention will deliver to lower speed devices thealerts in increasing bandwidth via SMS messaging, email alerts, audioalerts, video alerts and any combination thereof with links back to anyand all aspects of the WOI. Additionally, there are places and locationswhere streaming video is not appropriate, be it a bandwidth limitationor surroundings (e.g., meetings, formal occasions) would not make itsocially acceptable or simply because listening and viewing may not bepossible but reading would be perfectly normal and socially acceptable.

Embedded Software

The present invention further provides for a software program or Systemon a Chip (SOC) application that resides on a computational device atthe end-user's home, or at the broadcaster's premise or at a sharedfacility or service—such as Amazon's C3 and S3 network (“Cloud”)—thatwill monitor recorded TV programs, in progress TV programs, Internetbased Videos and streaming videos and recordings and extract the closedcaptions (CC) on a configurable basis; e.g., every N seconds, minutes orhours. The captions are retained on the user's device, pushed to theCloud (or any virtualized system) and/or both.

Preferably, the systems and methods of the present invention to collectand configure a personal dossier or clip comprised of one or morefragments from the words of interest from the captions and the extractedand configured CC. In one embodiment, the method steps preferablyinclude at least the following steps: extracting caption fragments froma broadcast; correctly sequencing the caption fragments by matchingfragment overlaps; eliminating redundancies; assembling the captionfragments into a single transcript; thereby providing a more completecaptions transcript from fragmented captions transcripts.

“Stitching” or assembling submitted video and/or audio segments fromvarious channels based on user interest—WOI or concepts—into a single ormultiple segments based on rules established by the application or theuser in a personal dossier or clip and allowing users to browse thevarious assembled dossiers private or publicly shared.

Delivering the stitched video and/or audio segments to an un-tethered ortethered device such as phone, TV, PC, radio, etc. The segments can thenbe shared using one or more of the available sharing platforms such asiTunes, Twitter, Facebook, etc. in a public or private channel.

Enabling feedback and rating of a segment or group of segments for thepurpose of ranking, augmenting or correcting the content by human orautomated means.

Analyzing and providing real-time or near real-time feedback to thebroadcaster or copyright owner of the content about usage, consumptionand interest about the WOI or concepts being broadcast thereby enablingthe content owner or broadcaster to better target various content toparticipating audiences.

Developing a SOC to accomplish the above and enabling it on PVRs, PCs,TVs, radios, phones and other electronic devices.

In a preferred embodiment, a software extension, SOC or plug-in (to aweb browser or an application) is incorporated in a audio oraudio-visual device or is provided through ‘add-on’ devices in networkedDigital Video Recorders (DVR) or Personal Video Recorders (PVR)systems—such as TiVo, Windows MCE or any of the Cable or Telephony basednetwork DVR and PVR systems, or a TV, mobile Telephone, standalonededicated device, web browser or computer application. This extensionwould enable the user to extract the CC on a configurable basis; e.g.,every n seconds, minutes or hours. The captions are retained on theuser's end device or pushed into the Cloud or both in order to leveragethe additional enhanced features of the entire environment.

In one embodiment of the present invention, a software program extractscaptions of various recording formats of various commercially availableTV Tuners formats, such as Microsoft's, ATI's, SnapStream, SageTV, etc.TV tuners located in an end user's home or a cloud configuration thattune the desired channels and record the programs of interest to theend-user. Tuners are available in many forms and tune un-encrypted andencrypted channels both in digital and non-digital formats, either instand-alone mode or added to a standard PC interface. The presentinvention systems and methods further include at least one database thatis capable of handling a large stream of incoming captions from multiplesources and segmenting the data access rights based on variousparameters such as but not limited to: personal channels, licensedchannels, free public channels, private channels, etc.

While there are both open source and licensed systems that deliver oneor more aspects of this capability, none allow for a very large scaledeployment (hundreds or thousands of channels from any source around theworld in any language) in a distributed or centralized manner using thesame components in a near real-time manner. All components of theapplication or service can run on a single system or many systems butappear as a single system or service.

Any Voice/Speech

The extraction of the captions would occur in for any transmission ofvoice or speech, including audio-only broadcasts such as radio-type overthe air, on the Internet or any connected network broadcasts utilizingtext to speech methods. The term “voice broadcast” is used herein toinclude any transmission, whether audio-visual or audio alone, thatbroadcast voice or speech, on any medium.

Advertisement Tag Code

The present invention provides systems and methods to encode and embed astream of bits that represent an Advertisement Tag Code (ATC) forproviding automatic electronic methods for collecting data about atleast one ATC and correlate collected data with additional sources ofdata. Benefits of methods and system of the present invention forapplications in advertising include providing a campaign managercapability to monitor automatically and electronically the effectivenessof a particular advertising campaign and to correlate such campaign withtraditional print media and internet media, mobile media campaigns. TheATC is placed in the VBI or closed captioned stream of a broadcast TVchannel, or in a live Internet video stream. The ATC need not beinserted in advertisements only, but in any type of broadcast such as anews broadcast or comedy shows. The ATC may be encrypted orun-encrypted, visible to the viewer (such as a QRcode or other code) orinvisible the user but recognizable by the automated systems, such asunique images, patterns, and the like.

The systems and methods of the present invention include at least oneadvertisement tag code (ATC) for electronically marking anadvertisement. Preferably, the ATC is provided at more than one point inthe data stream for a video advertisement, including once at thebeginning of an advertisement, or a begin tag, and also at the end of anadvertisement, or an end tag. The ATC may be either ‘open’ or‘encrypted’ and preferably includes an amount of information associatedwith predetermined factors, including anything that an advertiser orbroadcaster wishes to capture including but not limited to anadvertisement label, an intended advertisement market target, ademographic target, a television (TV) program, a time of advertisement,and a code, such as a general code or a proprietary code operable tolink the advertisement to a promotional campaign that is correspondinglylinked to the advertisement. A software application operable to collectdata and generate the at least one tag, and for extracting the at leastone tag, is provided within the system of the present invention.Preferably, an interactive website with graphic user interface isprovided for systems and methods of the present invention to allow amultiplicity of users to register for a web-based service for providingmethods for automated data stream ATC tagging. More particularly,registered users who have activated accounts via the interactive websiteplatform indicate or select and describe at least one advertisingcampaign for monitoring.

Within the context of the system and methods of the present invention,an advertisement agency or entity creating the advertisement(advertising users) provide and include an encoded or un-encoded streamof bits (or tag) in the beginning and end of every advertisement usingthe closed captioned technologies available today. The ATC or tag codeincludes any desired information that the advertiser wishes toencapsulate in the data. For example: that the advertisement played on aparticular region/ channel/ time slot/ day/ and that it played for aparticular amount of time.

A set of TV tuners or computing devices (in the event the ATCs are beingmonitored on the Internet by computers) would be located in targetlocations that will tune into the desired channels that the productmanagers wish to monitor and record the programs continuously based onan algorithm that is driven by the central service where and in nearreal-time, harvest the coded messages and update a data base orrepository, central or distributed, with the desired information.

Other Features

The software operable on the interactive web platform or softwareoperable on a remote computer device further include algorithms in thecloud platform or on the end user's computing device that are operablefor the following functions: capture the end user profile andpreferences with regards to WOI, modality of alerts, summarizationlevels of the CC and system housekeeping such as retention of recordedvideos and audios; format the incoming stream of captions to a more userfriendly and human readable format in any of the CC languages—captionsare not assumed to be English only—and to use various dictionaries toproperly format proper names, places, currencies, etc.; alert the userbased on any existing or future modality of interest such as mobiledevice, a computing device of any type, a browser plug-in, an RSS Readerof any type, a toolbar add-on to a browser of any kind or an operatingsystem feature capable of accepting one or more of the above modalities.

The present invention system and methods further include an indexingcapability that is capable of searching based on a variety of levelsranging from simple keywords, phrases, proximity of words, concepts,facets or ontological searches based on any publicly or privatelyavailable ontology; and a summarizer that is operable for summarizingthe full transcript of captions from a specific recording or partialcaptions of a specific recording at a varying degree of summarizationranging from 1% to 99% of the text—with zero and 100 percent being nosummary is necessary; a facility that is capable of detecting andaccordingly handling duplicate entries (e.g., same broadcast exists butis being re-broadcast at a different time or different channel) into thedatabase, ‘garbage’ (sometimes captions are garbled at the source anddue to transmission issues), offensive words (defined by the service orthe end user or both); a facility that is capable of standardizing anddetecting recording times across national and international boundariesin order to be able to retrieve and present the correct results forqueries into the CC database that will span multiple channels andmultiple time-zones. This facility also allows the service orapplication to integrate and ‘mash-up’ such information with a queryacross all indexed information from commercial search engines such asGoogle, Yahoo and Bing, Twitter, FaceBook and the like. The facilitythat is able to extract a segment, or all of the audio of a TVrecording, that matches the segment in the extracted CC where the WOIoccurred. The audio or video segments can be in one or more popularformats (e.g., mp3, mpeg) and can be optionally (based on the userprofile) combined into a single ‘clip’ or multiple ‘clips’, downloadedto a mobile device (e.g., iPhone or iPod Touch), integrated with apersonal media library (e.g., iTunes), or retained at the end-user'spremise or in the Cloud for future retrieval.

Furthermore, software of the present invention includes algorithms forgenerating analytics from the stored captions to answer questions suchas: what is being recorded, what is being searched, what modalities ofalerts users are choosing, ‘hot topics’ of the day, month or year, etc.;also, including algorithms operable for providing detailed informationof advertisement placements in a TV or Internet broadcast and provide acapability to back to the advertisement sponsor to link theadvertisement placement to the effectiveness of their promotioncampaigns on TV, Internet and other promotion campaigns; and optionallyincluding algorithms that enable comparison of competitive advertisementplacement campaigns to answer questions such as, by way of example andnot limitation: Where is Fidelity (or Schwab or E*Trade) advertising?What Shows? What Times? What Channels? How many? Preferably, thesoftware also includes algorithms operable for detecting whetheradvertisements were ‘clipped’ or shortened is provided for verifying thelength of the advertisement with the ‘Begin’ and End Tags of theadvertisement ATC.

Regarding system operation and methods thereof, the present inventionincludes operating at a computing device of an end user for thefollowing: installing a CC client software on the remote computingdevice of the end user; the software operable for automaticallyexecuting actions based upon selections input by the end user throughinput device(s) associated with the remote computing device, preferablyvia an interactive graphic user interface that is accessible via the webor other network; a cloud platform including a destination for captions,selectable or designatable by the end user; a database for storage ofextracted captions, either in memory on the remote computing device, onthe cloud platform, on removable memory device(s), or other data storagedevice or system; wherein the software application runs or operatesautomatically as a background task on the remote computer device foralways monitoring a recording folder for new recordings; andcombinations of these functions and/or components.

Additionally or alternatively, the present invention provides forsystems and methods operable from any networked device, providing forend user operation for the following: logon to a service or Internetsite and register the KOI and alert preferences; browse alerts; searchon KOI; display full, summary or clipping of transcript where the KOIoccurred; and combinations thereof.

Regarding back-end service operation for systems and methods of thepresent invention, the following functions are provided for set-up:set-up for TV tuners; connect tuners to TV and Internet providers, inparticular, as an option for centrally recorded channels; set-up forsoftware and system components on a centralized or distributed group ofcomputing and storage devices; set-up for network connectivity; andcombinations thereof.

FIG. 1 illustrates a schematic of the present invention systems andmethods, generally described as 100. The shaded parts of the diagram areexternal to the environment. Either a public or private networkdistribution mechanism is operable for the present invention. Anexecution computing device 10 is shown operable on a computer with TVtuners for functioning to capture video, extract & post captions to CCservice 30, whether cloud caption or other service. The mechanismprovides a TV tuner part of this environment, and a card inside a PC ortuner external to a PC, software monitors recordings, between about10-30 seconds, and user configurable for any length of time, looks fornew recordings, identifies recordings and captions correspondingthereto. Over time the software is able to delete recordings to conservestorage space. Once posted to the cloud or virtualized system, then theCC service components are orchestrated to store, search, alerted,summarize, etc. The link does not need to be an Internet link, since theentire system can be deployed in one box or in a distributed manneracross many systems.

Once at the cloud level or virtualized system, the system includes anarrangement that is distributed on one or more machines to scale. Acollector captures the CC (not shown) and saves in a database. A servicebus allows any system component to communicate and interact with anyother component in the system or service, e.g., summarizer can look forwhat is complete, alerter looks for profiles from users, and posts toalerting distribution module, etc., after confirming that it meetsclient profile, then provides a notice to client about what is recordedfrom shows or advertisements or other video being monitored. In thisembodiment, the search capability is built into the database but can bea separate index engine that resides either locally or another host evenexternal to the entire service and is constantly indexing the databasefor new information. A formatter (raw text captions come in alluppercase with an average of 5-6 words per line, prior art) attempts toformat what is being said, so that it provides a more human readabletext in free-form format. The harvester's role is for targetingcompanies, e.g., interested in product advertising, to respond or answerthe question of “who else is saying anything at this time related tothis product?” The system and methods of the present invention operateto harvest all data being said or communicated on the WOI, and link thedata to the point of interest (e.g., search engine results, social mediasites and the like to determine if product being mentioned anywhereelse, public or private resources). Significantly, the systems andmethods of the present invention provide for automated analysis of thedata including the WOI, wherein the analysis includes linking the targetmention results to other social media and digital media target mentionresults. More preferably, the analysis and automated linking of thetarget mention results to social media is applied for a predeterminedtime period. In this manner, correlation of the impact or value of thetarget mention results to response by an audience within a predeterminedtime and/or geography is provided. Thus, the present invention providessocial monitoring and assessment of target mention results, for examplein advertising or promotion of goods or services; a graphic userinterface or dashboard display may be provided to facilitate comparisonsand metrics for the analysis between the data including the WOI and thesocial media activity and/or response. By way of example and notlimitation, social media includes web-based sites for groups, such asFacebook, Twitter, and combinations thereof. Metrics are generated bythe cloud-based analysis of the data to determine how people relate tothe target mention results, such as for example with advertising, inparticular the analysis and metrics review related tweets or twitterfeeds and Facebook or social media text-based commentary with respect totime and content that was broadcast. Advantageously, since all CC aretime-stamped and date-stamped, the systems and methods provide forreal-time analysis. Harvesting twitter feeds or tweets and retweets withrespect to a content or a subject or WOI provides for analysis ofdistribution over time, data, and count (such as the number of tweets,retweets, etc. or social media mention).

The ATCs analytics database is including tags or currently captionedcommercials, to allow the analytics engine to determine how many times acompany is advertising on a specific channel, time, which company isadvertising on what stations and show, etc. A capability of theanalytics and alerting feature is to monitor, for example, if an ad ismentioned and a product manager is interested in knowing a potentialimpact of the ad with an analysis of web site traffic, links to thecompany's website, links to social media sites and analyze data toestablish any correlation between who is acting on the information beingadvertised. Preferably, the data is collected at the start of anadvertising campaign and analysis used to determine the effectiveness ofthe advertising campaign based upon the social media activitycorrelating in time with the WOI or target mention results.Additionally, a comparison with similar WOI in connection withcompetitive businesses may be provided with the analysis. Also, in thecase of Twitter or tweet analysis, retweets may be weighted overoriginal tweets since they amplify the impact or message propagationthrough that social media data. An automated survey application may befurther included for additional data to be used with the social mediadata to consider impact of the advertisement, WOI, and/or target mentionresults.

Social media platforms (e.g., Twitter, Facebook, Instagram, Snapchat,etc.) are a primary source of information for many users in recentyears. Social media tags on these social media platforms have helped tobuild communities of engaging discussion around particular news, events,persons, memes, topics, opinions, ideologies, etc. in the forms ofhashtag (e.g., # topic), @symbol, and the like.

For example, to help build a conversation around a subject, Twitterusers link their tweets to the subject by using hashtags. Hashtags canbe searched easily within social media sites to find out how many peoplehave been discussing a certain subject, how many times a certain subjecthas been discussed, and/or if the discussion is positive or negativeover a predetermined time and/or geographic area.

As the discussion grows and evolves, different hashtags can be used inrelation to a subject. The meaning of some hashtags may not be obviousgiven only the hashtag. The systems and methods of the present inventioncan identify all the hashtags related to a subject or words of interest,and provide for analysis including correlating hashtag discussions withtarget mention results in audio/video content sources. In oneembodiment, the present invention provides intelligence to businessowners in advertising or promotion of goods or services. In anotherembodiment, the present invention provides insights of the impact and/orinfluence of certain social and/or political events in a certain timeperiod and/or geographic area.

In one embodiment, the present invention is operable to create a quicksurvey based on captions and tags extracted from a video and/or audiosource, and solicit customer feedback via either a TV or a mobiledevice. In one embodiment, TVs are operable for interactive viewing. Alistener and/or viewer can click and select a tag in the closed captionsshown on a TV screen during a program to participate in a survey. Thesurvey through tags during a program captures relevant content, interestand demographics, and is more accurate and informative.

The surveys can be as simple or complex as the host desires. In oneembodiment, the survey is in an audio form. A podcast creator inserts atag (e.g., #survey) into the recording platform and creates a surveythat the podcast creator knows his listeners will not mind taking, forexample a three-question survey. Upon reaching the tag, the podcastplayer alerts the listener to a survey being present. The listener maychoose to pause the podcast to participate or continue listening and notparticipate in the survey. If the listener chooses to participate in thesurvey, the device connects to the appropriate survey and serves theuser either a textual or verbal survey. Upon completion of the survey,the podcast creator now has direct feedback about specific questionsregarding his or her survey. In another embodiment, the survey is in atext form. If the podcast's captions are being read on an electronicdevice, the user may click on the tag “#survey” to participate in thesurvey as if the user clicked on a hyperlinked “#survey.” The linkcontains all the information needed to serve the right survey to theparticular podcast. In another embodiment, the survey is embedded inclosed captions of a video. In the case of a hearing-impairedindividual, the viewer will see the #survey and may pause the podcast orvideo and participate in the survey.

In one embodiment, the present invention provides fact check for mediareports based on extracted captions from various news broadcast. It isassumed that news broadcasters vet their reports a lot more than fakenews writers. If a news story is reported by multiple stations, it isgiven a higher probability of truthiness than one that are not. In oneembodiment, the present invention provides an automated rating oftrustfulness to a news story based on corroboration of multipleindependent reputable and trusted sources including verifying timeline,prior stories, etc.

In one embodiment, the present invention extracts captions automaticallyfrom specific podcasts a user is interested in, and suggests similarpodcasts that have the same hosts or similar concepts being discussed.Links of similar podcasts are provided and subscription of thereof arepromoted. For example, a listener or a viewer has an interest in currentevents and is listening to or reading captions extracted from areal-time feed or archived feed. Key concepts in the real-time feed orarchived feed analyzed by the back-end platform include US-EUrelationships, President Trump's comments on NATO and the EU, DonaldTusk's comments about Vice Present Pence's commitment to InternationalOrder, security and the EU, and Vice President Pence's assuredcommitments to EU and NATO on his recent visit. The back-end platformthen suggests similar podcasts or captions from known sources (e.g.,television, radio, internet, newspapers) based on the listener'sinterest and key concepts from the current episode, and serves upadditional content that is likely to be of interest to the listener. Forexample, the back-end platform is operable to recommend published booksby a guest on the subject at hand, recommend additional podcasts orhosts where the guest was also featured, and provide links to otherpodcasts that address current events (e.g., from the Council on ForeignRelations, BBC, PBS Newshour, PRI's The World, etc.).

In one embodiment, the present invention provides automatic translationfor a script shown in a foreign language on a video source into alanguage that a user prefers (for example, his/her native language, or alangue he/she understands), and displays the translated script on a userdevice (for example, TVs, smart phones, and other portable devices witha display screen). For example, a protest sign written in a foreignlanguage in a news broadcast is automatically translated into a viewer'snative language. In one embodiment, the translated script stream is insync with the video stream and the extracted caption stream. In anotherembodiment, the translated script is embedded into the extracted captionstream.

The present invention preferably functions and is operable in a DVR 40and/or TV environment 50, as well as any computing device operable forvideo functions and capable of processing the embedded captions in avideo broadcast.

FIG. 2 is another schematic of an example embodiment of the presentinvention, demonstrating four local machines 10, including mobile localmachines, capturing and extracting captions and posting the extractedcaptions to a CloudCaptioned Service operating in a computing cloud.

In a preferred embodiment of the present invention, every part of atranscript is stored in the database. An email is forwarded to eachsubscriber according to the preferences for monitoring established bythe subscriber and authorized by the system. Data is retained in adatabase to enable additional deeper analytics for the purpose ofbusiness intelligence and creating decision support systems

The functions available remotely by the subscribers include: to browseand search transcripts; including delimiters such as, by way of exampleand not limitation, the date range, any words, all words, exact phrase,etc. as illustrated in the screen shot of FIG. 3. Preferably, the systemprovides access for the subscriber to review, preview, or see clippingsor portions of transcripts, or entire transcripts. A search can pull thesample where the search text occurs embedded in the captions. It canexpand the content in portions, including some additional data, but notfull transcript, or the entire transcript. As set forth herein, thesubscriber registers with the system to subscribe to keywords tomonitor, and receives an alert when it occurs; preferably, then aportion is provided, more than the alert; and then the option to see thefull transcript if entitled, and pay the content owner, as appropriate.

The systems and methods of the present invention also provide for a tagdatabase for advertisements, to include sounds, not just words. E.g.,Aflac advertisement might include “aflaaaac” the sound and not simplythe words. Such capability enables the execution environment torecognize intended advertisements or special tags without captions beingpresent in the broadcast. Then the system monitors shows oradvertisement on every show where the subscriber or advertisersadvertises and at what time the advertisement occurs. A preview of thesubscription to show results is provided automatically.

Also, preferably, an option under the graphic user interface of theinteractive website portal provides the option for selecting an outputformat as a spreadsheet for date, time, network, and show, which allowsa subscriber to quickly create macros for analytics to pivot arounddata. Additionally, as set forth hereinabove, the output format mayfurther include a dashboard or other GUI for presenting the data as wellas analysis thereof, including linking to live or archived social mediacontent from the likes of Facebook or Twitter.

Thus, the present invention includes a method for finding and accessingdesired audio content from audio content sources. The method stepsinclude providing a server with a processing unit, the server isconstructed, configured and coupled to enable communication over anetwork; the server provides for user interconnection with the serverover the network using a computing device positioned remotely from theserver; the server and personal computer running non-transitorycomputer-readable storage media with executable programs stored thereon;the personal computer monitoring a broadcast, the broadcast being anyvoice broadcast; the executable programs extracting captions from abroadcast in near real-time; aggregating the captions in a database;indexing the database content; searching the captions for a mention ofat least one target text, herein termed a target mention; analyzing theresults for desired content; indexing into the database to extract thedesired content; thereby providing a method for quickly finding andaccessing desired audio content from a large number of sources.

The method preferably further includes a local machine running anon-transitory computer-readable storage medium with an executableprogram stored thereon; the executable programs extracting the captions.The captions can be aggregated in one location or in a cloud computingsystem. The local machine's executable programs can be a system on achip application.

The method further includes analyses for determining the earned mediaand paid media of the at least one target and categorizing the at leastone target mentions into positive, negative, neutral and unknowncategories. The target mention results can be linked to other socialmedia and digital media target mention results, and therefore providefor social monitoring through social media usage. Preferably, theretrieved captions are retrieved from media selected from the groupconsisting of audio and/or video media.

Another method according to the present invention is a method formanaging communication through mass media; the method steps includemonitoring for target mentions; categorizing the target mentions intopositive, negative, neutral and unknown categories; linking the targetmentions in real-time to determine whether such mentions trigger a spikein social media; visualizing the results and analyzing for trends;responding to the media with interest with measured response based onthe results; measuring the impact of the response; thereby managingcommunication through mass media to increase mentions of a target. Themass media communication can be managed for different purposes,including public relations and brand management.

A method for preventing invalid captions from being submitted to aclosed caption database includes the method steps of authorizing andauthenticating linked devices; extracting captions from authenticatedlinked devices; thus preventing the submission of captions that are notpart of the broadcast. These method steps include at least the steps of:authorizing devices, authenticating linked devices; extracting captionsfrom authenticated linked devices; and preventing the submission ofcaptions that are not part of the broadcast. Security and authenticationare provided by private keys, shared keys, steganography and othermethods including a secret code that the server sends to each device,which code must be included with any uploaded caption segment, andcombinations thereof.

A method for extracting complete captions from fragmented audio or videocaptions includes the steps of extracting caption fragments from abroadcast; correctly sequencing the caption fragments by matchingfragment overlaps; eliminating redundancies; assembling the captionfragments into a single transcript; thereby providing a more completecaptions transcript from fragmented captions transcripts.

A system for extracting audio captions according to the presentinvention thus includes a server with a processing unit, a database, anda local machine tuned to at least one broadcast; the server constructed,configured and coupled to enable communication over a network; theserver and database and the server and local machine interconnected overthe network; the server and local machine running non-transitorycomputer-readable storage media with executable programs stored thereon;the executable programs of the local machine extracting captions fromthe broadcast and transmitting them to the server; the server executableprograms storing, indexing and retrieving the captions in and from thedatabase; thereby providing a system for local extraction of audiocaptions from a broadcast.

FIGS. 3-12 illustrate screen shots of various graphic user interfacesfor an interactive website according to one embodiment of the presentinvention. The various screen shots of website graphic user interfacesshow options for selecting search or browse transcripts, search allwords/any words/exact phrase, date range, shows to search, etc.Importantly, with the present invention systems and methods, it isprovided for functionality to automatically link between real timeadvertising on TV and then exploring web-based searching that followswithin a predetermined time. This provides for analytics that considermarketing and advertising conversion from viewers to searching onlinewithin a predetermined timeframe after it is shown on TV.

FIG. 3 shows an interface for retrieving captions for a specific show.FIG. 4 shows an interface for retrieving captions for a specific dateand time for the show selected in FIG. 3. FIG. 5 shows an interface forsearching shows. FIG. 6 shows the results of the query of FIG. 5. FIG. 7shows an interface for searching for advertisements subscriptions. FIG.8 shows the results of the query for FIG. 7, displayed in a timesequence. FIG. 9 shows the results of the query of FIG. 7, displayed byshow with count, including a graph display. FIG. 10 is the graph of FIG.9, enlarge for better visibility of details. FIG. 11 is a table andgraph showing query hits and count according to channel for the searchof FIG. 7. FIG. 12 is another table and graph showing query hits andcount according to channel for the search of FIG. 7.

Certain modifications and improvements will occur to those skilled inthe art upon a reading of the foregoing description. By way of exampleand not limitation, in addition to words of interest (WOI) or keywords,the present invention systems and methods include consideration ofconcepts associated with WOI, i.e., the concepts are considered as acontext within keywords or WOI but not identical to the WOI. For exampleif a person is interested in conflict situations in the Middle East, auser may only specify “Middle East conflict” as WOI but the system willbe capable of understanding the concept of Middle East conflict and willinclude fragments that discuss conflict in all the countries in theMiddle East (Israel, Lebanon, Iraq, Iran, Egypt, Jordan and Syria andnon-core participants such as US, Turkey, and many other countries)without the user explicitly specifying all the countries as specificWOI. The above-mentioned examples are provided to serve the purpose ofclarifying the aspects of the invention and it will be apparent to oneskilled in the art that they do not serve to limit the scope of theinvention. All modifications and improvements have been deleted hereinfor the sake of conciseness and readability but are properly within thescope of the present invention.

The invention claimed is:
 1. A method for finding and analyzing target content, comprising: providing at least one device and a cloud-based platform, wherein the cloud-based platform comprises at least one server and wherein the at least one device communicates with the cloud-based platform over at least one network; the at least one device extracting captions of the audio and video content sources; the cloud-based platform receiving extracted captions from the at least one device; the cloud-based platform searching the extracted captions for at least one keyword relating to the target content, thereby creating search result data; the cloud-based platform harvesting social media data relevant to the target content; and the cloud-based platform analyzing the target content based on the search result data in correlation with the social media data.
 2. The method of claim 1, further comprising the cloud-based platform extracting audio and video segments relevant to the target content from the audio and video sources; and the cloud-based platform delivering extracted audio and video segments to the at least one device.
 3. The method of claim 2, further comprising sharing extracted audio and video segments on at least one sharing platform selecting from the group consisting of iTunes, Twitter, Facebook and other social media or digital media platforms.
 4. The method of claim 3, further comprising receiving feedback and/or rating of the extracted audio and video segments on the at least one sharing platform.
 5. The method of claim 4, further comprising analyzing the target content based on the feedback and/or rating in correlation with the search result data and the social media data.
 6. The method of claim 1, further comprising identifying at least one social media tag relevant to the at least one keyword for the target content.
 7. The method of claim 6, wherein the at least one social media tag comprises at least one hashtag relevant to the at least one keyword for the target content.
 8. The method of claim 1, wherein the social media data is harvested in a predetermined time period and/or geographic area.
 9. The method of claim 1, wherein the social media data is harvested from at least one social media site selected from the group consisting of Twitter, Facebook, Instagram, Snapchat, and other web-based sites for groups.
 10. The method of claim 1, further comprising the cloud-based platform analyzing the target content in real time or near real time.
 11. The method of claim 1, further comprising a graphic user interface displaying and facilitating comparisons and metrics between the search result data and the social media data.
 12. The method of claim 1, further comprising authorizing and authenticating the at last one device.
 13. A method for finding and analyzing target content, comprising: providing at least one device and a cloud-based computing system, wherein the at least one device communicates with the cloud-based computing system over at least one network; the at least one device extracting captions of at least one audio or video; the cloud-based computing system receiving extracted captions from the at least one device; the cloud-based computing system searching the extracted captions for target content based on user profile and preferences, thereby creating search result data; the cloud-based computing system harvesting social media data relevant to the target content in a predetermined time period; and the cloud-based computing system analyzing the target content based on the social media data correlated in time with the search result data.
 14. The method of claim 13, further comprising: the cloud-based computing system extracting audio or video segments relevant to the target content; the cloud-based computing system delivering at least one alert regarding extracted audio or video segments to the at least one device; and formatting the extracted captions to a more human readable text in free-form format.
 15. The method of claim 13, wherein the user profile and preferences comprise words of interest, modality of alerts, summarization levels of the extracted captions, and system housekeeping.
 16. The method of claim 13, wherein the at least one alert is via Short Message Service (SMS) messaging, email alerts, audio alerts, video alerts, or any combination thereof
 17. A method for analyzing an advertisement campaign, comprising: providing at least one device and a cloud-based computing system, wherein the at least one device communicates with the cloud-based computing system over at least one network; the at least one device extracting captions of at least one audio or video related to the advertisement campaign; the cloud-based computing system receiving extracted captions from the at least one device; the cloud-based computing system searching the extracted captions for at least one keyword related to the advertisement campaign, thereby creating search result data; the cloud-based computing system monitoring social media activities related to the advertisement campaign, thereby creating social media data; and the cloud-based computing system analyzing effectiveness of an advertisement campaign based on the social media data correlated with the search result data.
 18. The method of claim 17, further comprising: generating at least one advertisement tag code (ATC) for the advertisement campaign; marking the at least one audio or video related to the advertisement campaign with the at least one ATC; and extracting audio or video segments from the at least one audio or video related to the advertisement campaign.
 19. The method of claim 18, wherein the at least one ATC comprises information associated with predetermined factors selecting from the group consisting of an advertisement label, an intended advertisement market target, a demographic target, a television program, a time of advertisement, and a code; and wherein the code is operable to link the at least one audio or video to the advertisement campaign.
 20. The method of claim 18, further comprising performing surveys related to the advertisement campaign, thereby creating survey data; and analyzing the effectiveness of the advertisement campaign based on the survey data correlated in time with the social media data. 